System and method for selecting sponsored images to accompany text

ABSTRACT

A system for selecting an image to accompany text from a user in connection with a social media post. The system includes receiving text from the user; identifying one or more search terms based on the text; identifying candidate images from images in one or more image databases using the search terms, where the candidate images comprise a sponsored image; presenting one or more candidate images to the user, where the sponsored image is presented preferentially compared to other candidate images; receiving from the user a selected image from the one or more candidate images; generating the social media post comprising the selected image and the user-submitted text; and transmitting the social media post for display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.14/508,088, filed Oct. 7, 2014, now U.S. Pat. No. ______ which is aContinuation In Part of U.S. Non-Provisional application Ser. No.14/478,037 filed Sep. 5, 2014, which is incorporated by reference in itsentirety.

BACKGROUND

In the age of social media, multimedia messages containing both text andimages may convey thoughts and ideas to others, including social orbusiness contacts or to a larger audience. Selecting an appropriateimage to pair with text provided by a user may be a challenging andlaborious process, given the large number of images available to aninternet user.

Other methods of searching for images require the user to select oridentify discrete search terms to find the image and/or search amongstall images on the internet. As users become accustomed to the process ofcreating a multimedia message where the image is suggested by a service,opportunities arise for customization in ways that could benefitpublishers, photographers, advertisers, or anyone with a financial,aesthetic, or editorial interest in influencing the images suggested tothe user while creating their multimedia message.

Accordingly, an automated method for suggesting and choosing images topair with a user's text may be desirable. A service to publish and sharesuch thoughts, and to track them so that images for future thoughts maybe better chosen, may be similarly desirable.

SUMMARY OF THE INVENTION

A system for selecting an image to accompany text from a user inconnection with a social media post. The system includes receiving textfrom the user; identifying one or more search terms based on the text;identifying candidate images from images in one or more image databasesusing the search terms, where the candidate images comprise a sponsoredimage; presenting one or more candidate images to the user, where thesponsored image is presented preferentially compared to other candidateimages; receiving from the user a selected image from the one or morecandidate images; generating the social media post comprising theselected image and the user-submitted text; and transmitting the socialmedia post for display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of creation of a post in accordance with oneaspect of the present invention.

FIG. 2 is a flow chart of a configuration process in accordance with oneaspect of the present nvention.

FIG. 3 is a flow chart of an image identification and selection processin accordance with one aspect of the present invention.

FIG. 4 is a flow chart of an image identification and selection processin accordance with one aspect of the present invention.

FIG. 5 is a flow chart of an image identification and selection processin accordance with one aspect of the present invention.

FIG. 6 is a diagram displaying a computer on which the system and methodof the present disclosure may be implemented according to an aspect ofthe present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a flow chart of creation of a post in accordance with oneaspect of the present invention. A “post” may be text displayed alongwith media, where the text and/or media may be supplied by the user. Themedia may be an image, video, and/or audio. The post may includeassociated search terms that may be supplied by the user and/orgenerated by the system. Some or all of the post may be created by auser and may be published so as to be viewable by the user and possiblyothers. In Step 1 of the post creation process shown in FIG. 1, the usermay input text. In one aspect, the user may optionally also input one ormore search terms relating to the post. In another aspect, the systemmay select search terms based on the text, and the user may optionallyedit or add search terms. Each search term may be one or more wordsincluded in or associated with the post. In Step 2, based on one or moreof the text, the search terms, and posts by the user and/or other users,the system may identify candidate images to accompany the post. Thisidentification process is described in more detail below. In Step 3, theuser may select one of the candidate images or may select or inputanother image. In Step 4, the user finalizes the post and publishes itusing the system.

FIG. 2 is a flow chart of a configuration process wherein a corpus ofposts are built that may be used to identify one or more candidateimages to associate with text in future posts. In Step 11, the systemmay access or compile posts, for example posts for which the system hastext and search terms. The text and corresponding search terms for apost may be stored in a document. In Step 12, each post may be tokenizedinto the individual words in the post. Step 12 may include the removalof stop words (which may include certain short or less critical wordssuch as “the,” “is,” “at,” “which,” and “on”) and/or may include thenormalization of ASCII emoticons in the post. In one aspect of theinvention, the system may perform stemming to reduce words to their stem(or base, or root) form for better evaluation. In Step 13, each word maybe assigned a positive integer value. In one aspect, the words may beassigned an integer value without regard to the case of the letters withwhich the word was written (e.g. Sample, sample, and SAMPLE wouldreceive the same integer value). In Step 14, each document d may bemapped to a vector v=f(d) in an N-dimensional vector space V, where Nmay be, in one aspect, the number of distinct positive integers in thedocument or any set of unique values could be used as a vector space. fmay be a function that receives a sentence/document d and outputs avector where the value of each word b_i may be the number of times b_iappears in d multiplied by the fraction of 1/(the number of times b_iappears in the entire corpus). This process is known as bag-of-wordsrepresentation, weighted by TF-IDF, also known as term frequency-inversedocument frequency. This process helps identify unusual or less-commonwords that may be indicative or unique thoughts in the text, whileignoring more commonly occurring words.

Once the corpus is built, it may be used to find prior posts similar tothe new post currently being submitted to the system, so that similarimages can be located and presented to the user creating the new post.FIG. 3 is a flow chart depicting the image identification and selectionprocess after text for a new post has been submitted by the user. InStep 21, the text of the post may be sent from the user client to theserver. In one aspect, the system, including the methods depictedherein, may run on a server, and the server may periodically communicatewith a plurality of clients allowing users to read and write posts. InStep 22, the text of the submitted post may be mapped to vector space Vto identify similar posts. This may be the same vector space asdiscussed with reference to Step 14 in FIG. 2. In Step 23, the systemsearches for similar posts in the corpus. In accordance with one aspectof the invention, this list may include a predetermined number of mostsimilar posts, e.g., 100, or may use another threshold determiningfactor to identify similar posts. The similar posts may be limited tothose with known search terms. In accordance with one aspect, similarposts are identified using cosine similarity as the distance function(i.e., the measure of the angle between the two vectors representing thetwo posts). In Step 24, the system determines whether a threshold numberof similar posts exist. If not (“No”, Step 24), then in Step 32 thesystem may perform a sentiment analysis on the post text that willoutput a sentiment of the content of the text, in one aspect, if thesentiment analysis deems the text to have a positive emotionalsentiment, then the sentiment analysis may output positive emotionalterms such as “sunlight” and “smile”. if the sentiment analysis deemsthe text to have a negative emotional sentiment, then the sentimentanalysis may output negative emotional terms such as “cold” and “dark”.The output of the sentiment analysis may be saved as search terms tocorrespond with the post. Once those search terms are known, the systemmay perform a search on one or more image provider databases using thesearch term(s) in Step 30, and the resulting candidate images arereceived in Step 31.

If similar, the results from a similarity search exist (“Yes”, Step 24),then in step 25, a list of the most similar posts with known searchterms may be retrieved. Once the list of similar posts is received, theleast similar posts are removed from the list in Step 26. This may bedone by generating a similarity score. In one aspect, posts with asimilarity score below a predetermined threshold, e.g., less than 0.5,will be removed. In another aspect, relative thresholds may be used suchthat posts are only considered similar if they are within a relativethreshold of similarity to the best matching similar post. In anotheraspect, a post can be considered to have a greater degree of similarityto another post based on multiple appearances of the same search term.For each post with a similarity score over the threshold, the searchterms of that post, which may have been defined by the user whopublished the post or by another entity, are retrieved in Step 27. InStep 28, the list of search terms may be sorted by the number of timeseach search term occurs in all of the remaining similar posts and thelist may be de-duplicated. In Step 29, a number (e.g., four) of searchterms present most often in the most similar posts may be returned. Oncethose search terms are selected, a search may be run on image providerdatabases using the search term(s) with the highest similarity score(s)in Step 30, and the resulting candidate images are returned in Step 31.

Candidate images may be displayed to the user on the client device,during the search and/or after the search is completed. The user maypreview a candidate image with the user's accompanying text. In oneaspect, the system may track the use of images. The system may eliminateas candidates certain images, for example, images that have beenselected by users for use with posts more than a predetermined number oftimes, within a predetermined time period and/or geographic region.

In one aspect, the system presents the user with one or more alternativecandidate images from which the user may further choose. Alternativecandidate images may be presented in the order they are returned fromthe image service. In another aspect, the alternative candidate imagesmay be derived from the search terms and may be sorted bycharacteristics such as accessibility, size, and/or speed of retrieval.For example, images may be retrieved faster via HTTP than HTTPS, andsmaller images may load faster than larger images. Accessibility may bedetermined, for example, by retrieving the image, by sending a “HEAD”HTTP command, or by making a TCP socket connection to the server.

Once the user selects an image, the user may decide to publish the post,after which the post may be added to the corpus of posts in accordancewith FIG. 2. The image that was published with the post, or a link(e.g., a URL) thereto, may also be stored. As discussed above, thesource image (and/or a link) may also be stored for the purposes ofprohibiting repeated frequent use of that image, e.g., by region and/orby user.

FIG. 4 is a flow chart depicting another image identification andselection process after text for a new post has been submitted by theuser. In Step 41, the text of the post may be sent from the user clientto the server. In one aspect, the system, including the methods depictedherein, may run on a server, and the server may periodically communicatewith a plurality of clients allowing users to read and write posts. InStep 42, the text of the submitted post may be mapped to vector space Vto identify similar posts. This may be the same vector space asdiscussed with reference to Step 14 in FIG. 2, In Step 43, the systemsearches for similar posts in the corpus. In accordance with one aspectof the invention, this list may include a predetermined number of mostsimilar posts, e.g., 100, or may use another threshold determiningfactor to identify similar posts. The similar posts may be limited tothose with known search terms, in accordance with one aspect, similarposts are identified using cosine similarity as the distance function(i.e., the measure of the angle between the two vectors representing thetwo posts). In Step 44, the system selects search terms from the similarposts. In so doing, the system may not select as search terms certainwords deemed unhelpful or non-informational. Once search terms areselected, a search may be run on image provider databases using thesearch term(s) with the highest similarity score(s) in Step 30, and theresulting candidate images are returned in Step 31.

FIG. 5 is a flow chart depicting another image identification andselection process after text for a new post has been submitted by theuser. As described above, the user may input or select and/or the systemmay identify search terms based on text submitted by the user, as shownin Step 51 of FIG. 5. The system may then search databases in Steps 52and 53. Step 52 may occur before, after, in parallel with, or instead ofthe search in Step 53, In Step 53, the system searches one or more imagedatabases as described above. In Step 52, the system searches asponsored database populated with images provided by, approved by orotherwise associated with third-party advertisers. Each image may beassociated with one or more search terms. In another aspect, instead ofor in addition to searching based on search terms, the system may searchbased on one or more other factors including, geographic location of theuser, time of day, week, month or year, calendar event, geographicand/or temporal data relating to nearby activities or events, anduser-dependent properties including information associated with previousposts from the user.

In Step 54, the system selects candidate images from databases, thensends those candidate images to the user at Step 55. In one aspect,sponsored candidate images may be displayed to the user first, ahead of,in a preferential position or in a specified proportion compared tonon-sponsored candidate images. In one non-limiting example, images inthe databases may have corresponding information, such as geographic ortemporal information. Continuing that example, when the system receivestext for a post by a user located near a professional sporting arenawhile a sporting event is taking place, the system may select ascandidate images those images relating to the sporting event instead ofor in additional to images based on search terms. In Step 56, the systemreceives information identifying a selected image from the user. In Step57, the system creates and publishes a post as described above. In Step58, the system may receive responses to the post from the creating useror other users. Responses may include indications of attention orinterest from other users. Responses may include indications that a user“likes” or “favorites” the post. Responses may include “Reply posts”created by users that are linked to the original post, personalmessages, and requests to chat with the posting user. Responses mayinclude “shares”, where users share the post with others, e.g, viaemail, instant or text message, social media or other outlet.

At Step 59, the system may perform accounting related to the sponsoredimages and posts containing those images. For example, the system maytrack instances when sponsored images are presented as candidates tousers, instances when sponsored images are selected for use in posts,responses to posts containing sponsored images, and the number of timesthose posts are viewed via web and/or mobile. Such tracked informationmay be considered advertising impressions and may be used to invoiceadvertisers. The system may track the number of keywords from acampaign. A campaign may include one or more images associated with anadvertiser and may include one or more keywords associated with each ofthe one or more images. The system may report posts containing sponsoredimages to advertisers, and advertisers may receive rights to reproducethose posts.

In one aspect, the system may treat reply posts similarly to originalposts and perform the selection steps described above. In anotheraspect, the system may treat reply posts differently from originalposts. In that aspect, when a second user creates a reply post to anoriginal post, where the original post uses a sponsored image, then thesystem may automatically suggest one or more sponsored images ascandidate images to the second user for use in the reply post. In oneaspect, the system may suggest one or more candidate images to the replypost where the candidate images are associated with the image used inthe original post and/or with a sponsor or campaign associated with theimage used in the original post.

FIG. 6 shows a general purpose computer on which the system and methodof the present disclosure may be implemented. The computer system 900may execute at least some of the operations described above. Computersystem 900 may include processor 910, memory 920, storage device 930,and input/ output devices 940. Some or all of the components 910, 920,930, and 940 may be interconnected via system bus 950. Processor 910 maybe single or multi-threaded and may have one or more cores. Processor910 may execute instructions, such as those stored in memory 920 or instorage device 930. Information may be received and output using one ormore input/output devices 940.

Memory 920 may store information and may be a computer-readable medium,such as volatile or nonvolatile memory. Storage device 930 may providestorage for system 900 and may be a computer-readable medium. In variousaspects, storage device 930 may be a flash memory device, a floppy diskdevice, a hard disk device, an optical disk device, or a tape device.

Input/output devices 940 may provide input/output operations for system900. Input/output devices 940 may include a keyboard, pointing device,and microphone. Input/output devices 940 may further include a displayunit for displaying graphical user interfaces, speaker, and printer.External data may be stored in accessible external databases 960.

The features described may be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations thereof. The apparatus may be implemented in a computerprogram product tangibly embodied in an information carrier, e.g., in amachine-readable storage device or in a propagated signal, for executionby a programmable processor; and method steps may be performed by aprogrammable processor executing a program of instructions to performfunctions of the described implementations by operating on input dataand generating output.

The described features may be implemented in one or more computerprograms that are executable on a programmable system including at leastone programmable processor coupled to receive data and instructionsfrom, and to transmit data and instructions to, a data storage system,at least one input device, and at least one output device. A computerprogram may include set of instructions that may be used, directly orindirectly, in a computer to perform a certain activity or bring about acertain result. A computer program may be written in any form ofprogramming language, including compiled or interpreted languages, andit may be deployed in any form, including as a stand-alone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment.

Suitable processors for the execution of a program of instructions mayinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors ofany kind of computer. Generally, a processor may receive instructionsand data from a read only memory or a random access memory or both. Sucha computer may include a processor for executing instructions and one ormore memories for storing instructions and data. Generally, a computermay also include, or be operatively coupled to communicate with, one ormore mass storage devices for storing data files; such devices includemagnetic disks, such as internal hard disks and removable, disks;magneto-optical disks; and optical disks. Storage devices suitable fortangibly embodying computer program instructions and data may includeall forms of non-volatile memory, including by way of examplesemiconductor memory devices, such as EPROM, EEPROM, and flash memorydevices; magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory may be supplemented by, or incorporated in, ASICs(application-specific integrated circuits).

To provide for interaction with a user, the features may be implementedon a computer having a display device such as a CRT (cathode ray tube)or LCD (liquid crystal display) monitor for displaying information tothe user and a keyboard and a pointing device such as a mouse or atrackball by which the user may provide input to the computer.

The features may be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system may be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks may include, e.g., a LAN, a WAN, andthe computers and networks forming the Internet.

The computer system may include clients and servers. A client and servermay be remote from each other and interact through a network, such asthe described one. The relationship of client and server may arise byvirtue of computer programs running on the respective computers andhaving a client-server relationship to each other.

Numerous additional modifications and variations of the presentdisclosure are possible in view of the above teachings. It is thereforeto be understood that within the scope of the appended claims, thepresent disclosure may be practiced other than as specifically describedherein.

What is claimed is:
 1. A system for selecting an image to accompany textfrom a user in connection with a social media post, comprising aprocessor and a tangible storage medium, said tangible storage mediumcomprising code executable by the processor for performing stepscomprising: a) receiving text from the user; b) identifying one or moresearch terms based on the text; c) identifying a set of candidate imagesfrom images in one or more image databases using the search terms, wherethe set of candidate images comprises a sponsored image; d) eliminatingan image from the set of candidate images based on an eliminationfactor; e) presenting at least a subset of the set of candidate imagesto the user, where the sponsored image is presented preferentiallycompared to other candidate images; f) receiving from the user aselected image from the set of candidate images; g) generating thesocial media post wherein the social media post is of a similar size andof a similar shape as the selected image, and wherein the social mediapost comprises the text displayed superimposed on the selected image;and h) transmitting the social media post for display.
 2. The system ofclaim 1, wherein step f) further comprises storing information relatingto the selected image including one or more of an image identifier,time/date of use, use occurrence, and geographic location of the user.3. The system of claim 1, wherein the elimination factor comprises oneor more of time of use of the image, date of use of the image, frequencyof use of the image, and geographic location of the user.
 4. The systemof claim 1, wherein the user-submitted text further comprises one ormore search terms and step b) comprises identifying those search terms.5. The system of claim 1, wherein step b) comprises generating searchterms based on the user-submitted text.
 6. The system of claim 1,wherein step e) further comprises presenting a predetermined number ofsponsored images.
 7. The system of claim 1, wherein step e) furthercomprises presenting sponsored images in a predetermined proportion toother images.
 8. The system of claim 1, wherein the selected image isthe sponsored image, and further comprising step i) accounting views ofthe social media post by other users in connection with the sponsoredimage, a campaign, or a third-party associated with the sponsored image.9. The system of claim 1, wherein the selected image is the sponsoredimage, and further comprising step i) receiving responses to the socialmedia post and accounting those responses in connection with thesponsored image image, a campaign, or a third-party associated with thesponsored image.
 10. The system of claim 9, wherein the responsecomprises a like, favorite, reply, personal message, or request to chatwith the user.
 11. The system of claim 1, wherein the selected image isthe sponsored image and further comprising: i) receiving from a seconduser a request to reply to the social media post and receiving text fromthe second user; and j) presenting one or more candidate images to thesecond user, whesponsored re a second spon image is presentedpreferentially compared to other candidate images, and wherein thesecond sponsored image is linked to a sponsor or campaign associatedwith the sponsored image.
 12. The system of claim 11, furthercomprising: k) receiving from the second user a second selected imagefrom the one or more candidate images; l) generating the social mediareply post comprising the second selected image and the seconduser-submitted text; and m) transmitting the social media reply post fordisplay.
 13. A system for selecting an image to accompany text from auser in connection with a social media post, comprising a processor anda tangible storage medium, said tangible storage medium comprising codeexecutable by the processor for performing steps comprising: a)receiving text and geographic location information from the user; b)identifying one or more search terms based on the text; c) identifyingcandidate images from images in one or more image databases using thesearch terms and one of the geographic location information and thetime/date, where the candidate images comprise a sponsored image; d)presenting one or more candidate images to the user, where the sponsoredimage is presented preferentially compared to other candidate images; e)receiving from the user a selected image from the one or more candidateimages and storing information relating to the selected image includingone or more of an image identifier, time/date of use, use occurrence,and geographic location of the user; f) generating the social media postwherein the social media post is of a similar size and of a similarshape as the selected image, and wherein the social media post comprisesthe text displayed superimposed on the selected image; and g)transmitting the social media post for display.
 14. The system of claim13, wherein the selected image is the sponsored image and furthercomprising: h) receiving from a second user a request to reply to thesocial media post and receiving text from the second user; and i)presenting one or more candidate images to the second user, where asecond sponsored image is presented preferentially compared to othercandidate images, and wherein the second sponsored image is linked to asponsor or campaign associated with the sponsored image.
 15. The systemof claim 14, further comprising: j) receiving from the second user asecond selected image from the one or more candidate images; k)generating the social media reply post comprising the second selectedimage and the second user-submitted text; and m) transmitting the socialmedia reply post for display.